Skip to content

Add BFCL sweep milestone gates - #39

Closed
SonAIengine wants to merge 1 commit into
codex/bfcl-tail-hardcase-liftfrom
codex/bfcl-sweep-milestone-gates
Closed

Add BFCL sweep milestone gates#39
SonAIengine wants to merge 1 commit into
codex/bfcl-tail-hardcase-liftfrom
codex/bfcl-sweep-milestone-gates

Conversation

@SonAIengine

Copy link
Copy Markdown
Owner

Summary

  • add a default xgen-0.27 milestone gate to BFCL model sweep artifacts
  • include per-category sweep rows and category repeat groups so parallel_multiple bottlenecks are visible without hand-calculation
  • document how to use the gate before deciding whether to run another full expensive benchmark

Why

Full BFCL model runs can take hours. This makes the sweep artifact answer the immediate research question: did retrieved k=5 preserve enough of row-source performance for the 0.27 XGEN target, and is parallel_multiple still the blocking category?

Validation

  • poetry run ruff check .
  • poetry run ruff format --check .
  • poetry run pytest tests/ -q -> 949 passed, 3 skipped
  • CLI smoke: poetry run python -m benchmarks.bfcl_tool_selection.sweep --categories simple_python --limit 1 --top-ks 5 --tool-sources retrieved --model fake --llm-url http://127.0.0.1:9/v1 --timeout 1

@SonAIengine

Copy link
Copy Markdown
Owner Author

Closing this internal stacked PR as superseded by the current main branch. The branch is preserved for history.

@SonAIengine SonAIengine closed this Aug 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant